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1 . (Currently amended) A method of recovering a three-dimensional scene from two- 
dimensional images, the method comprising: 

providing a sequence of images; 

dividing the sequence of images into segments , the segments comprising groups of frames ; 
performing three-dimensional reconstruction for each segment individually; and 
combining the three-dimensional reconstructed segments together to recover a three- 
dimensional scene for the sequence of images. 

2. (Original) The method of claim 1 wherein performing includes creating virtual key 
frames for each of the segments, wherein the virtual key frames are only a subset of the images in a 
segment but are a representation of all of the images in that segment. 

3. (Original) The method of claim 1 wherein the images contain feature points and a 
number of images included in a segment is based upon a number of feature points tracked between 
the images in that segment. 

4. (Original) The method of claim 1 wherein performing further includes: 
performing a two-frame structure-from-motion algorithm to create a plurality of local models 

for each segment; and 

combining the plurality of local models by eliminating scale ambiguity. 

5. (Original) The method of claim 4 further comprising: 

bundle adjusting the combined local models to obtain a partial three-dimensional model for 
each segment; 

extracting virtual key frames from the partial three-dimensional model, wherein the virtual 
key frames include three-dimensional coordinates for the images and an associated uncertainty; and 
bundle adjusting all segments to obtain a complete three-dimensional model. 



6. (Original) The method of claim 1 further including: 
identifying feature points in the images; 
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estimating three-dimensional coordinates of the feature points; and 

estimating a camera rotation and translation for a camera that captured the sequence of 

images. 

7. (Original) The method of claim 1 wherein combining includes performing a non- 
linear minimization process across the different segments through bundle adjustment. 

8. (Original) A computer-readable medium having computer-executable instructions for 
performing the method recited in claim 1. 

9. (Original) A method of recovering a three-dimensional scene from two-dimensional 
images, the method comprising: 

identifying a sequence of two-dimensional frames that include two-dimensional images; 
dividing the sequence of frames into segments, wherein a segment includes a plurality of 

frames; 

for each segment, encoding the frames in the segment into at least two virtual frames that 
include a three-dimensional structure for the segment and an uncertainty associated with the 
segment. 

1 0. (Original) The method of claim 9 wherein dividing includes: 
identifying a base frame; 

identifying feature points in the base frame; and 

defining the segments such that every frame in a segment has at least a predetermined 
percentage of feature points identified in the base frame. 

1 1 . (Original) The method of claim 9 wherein the segments vary in length and wherein 
the length is associated with the number of frames in the segment. 

12. (Original) The method of claim 9 further including: 
identifying feature points in the sequence of two-dimensional frames; 
estimating three-dimensional coordinates for the feature points; and 
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13. (Original) The method of claim 12 wherein estimating the three-dimensional 
coordinates includes applying a two-frame structure-from-motion algorithm to the sequence of two- 
dimensional frames. 

14. (Original) The method of claim 9 further including: 
dividing a segment into multiple frame pairs; 

applying a two-frame structure-from-motion algorithm to the multiple frame pairs to create a 
plurality of local models; and 

scaling the local models so that they are on a similar coordinate system. 

15. (Original) The method of claim 14 wherein each of the multiple frame pairs includes 
a common base frame and one other frame in the segment. 

16. (Original) The method of claim 15 further including interpolating frames between the 
multiple frame pairs. 

1 7. (Original) The method of claim 9 wherein encoding includes: 

choosing at least two frames in the segment that are at least a threshold number of frames 

apart; 

for each of the at least two chosen frames, projecting a plurality of three-dimensional points 
into a corresponding virtual frame; and 

for each of the at least two chosen frames, projecting an uncertainty into the corresponding 
virtual frame. 

18. (Original) The method of claim 9 further including bundle adjusting the virtual 
frames from the segments to create a three-dimensional reconstruction. 

19. (Original) The method of claim 9 further including identifying feature points in the 
frames by using motion estimation. 
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20. (Original) The method of claim 19 wherein the motion estimation includes: 
creating a template block in a first frame including a feature point and neighboring pixels 

adjacent the feature point; 

creating a search window used in a second frame; and 

comparing an intensity difference between the search window and the template block to 
locate the feature point in the second frame. 

21 . (Original) The method of claim 9 wherein at most two virtual frames are used. 

22. (Original) A computer-readable medium having computer-executable instructions for 
performing the method recited in claim 9. 

23. (Currently amended) A method of recovering a three-dimensional scene from a 
sequence of two-dimensional frames, comprising: 

(a) segmenting the sequence of two-dimensional frames into segments, the segments 
comprising groups of frames ; 

(b) identifying feature points in at least a first base frame in a first segment; 

(c) analyzing a second frame in the segment to identify the feature points in the second 

frame; 

(d) determining whether a threshold number of feature points from the base frame are 
identified in the second frame; 

(e) if a threshold number of feature points are identified in the second frame, adding the 
second frame to the segment; and 

(f) repeating (c) through (e) for subsequent frames until the number of feature points in a 
frame falls below the threshold number. 

24. (Original) The method of claim 23 further including designating a frame that falls 
below the threshold number as a base frame in a second segment and repeating (b) through (e) for 
the second segment. 
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25. (Original) The method of claim 23 further including performing motion estimation to 
identify the feature points. 



26. (Original) The method of claim 23 further including using corners as the feature 

points. 

27. (Original) The method of claim 23 wherein the number of frames varies between 
segments. 



28. (Original) The method of claim 23 further including creating two virtual key frames 
per segment. 

29. (Original) The method of claim 28 further including bundle adjusting the virtual key 
frames of all the segments to obtain a three-dimensional reconstruction. 

30. (Original) A computer-readable medium having computer-executable instructions for 
performing the method recited in claim 23. 



31. (Currently amended) In a method of recovering a three-dimensional scene from a 
sequence of two-dimensional frames, an improvement comprising dividing a long sequence of 
frames into segments , the segments comprising groups of frames, and reducing the number of frames 
in each segment by representing the segments using between two and five representative frames per 
segment, wherein the representative frames are used to recover the three-dimensional scene and 
remaining frames are discarded so that the three-dimensional scene is effectively compressed. 



32. (Original) The method of claim 31 wherein each of the representative frames have an 
uncertainty associated therewith. 



33. (Original) The method of claim 31 wherein the long sequence includes over 75 

frames. 
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34. (Original) The method of claim 31 wherein dividing the long sequence into segments 
includes identifying a base frame and tracking feature points between frames in the sequence and the 
base frame and ending a segment whenever a frame does not contain a predetermined threshold of 
feature points that are contained in the base frame. 

35. (Original) The method of claim 31 further including performing a two-frame 
structure-from-motion algorithm on each of the segments to create a partial model. 

36. (Currently amended) A computer-readable medium having computer-executable 
instructions for performing a method comprising: 

providing a sequence of two-dimensional frames; 

dividing the sequence into segments , the segments comprising groups of frames ; 

calculating a partial model for each segment that includes three-dimensional coordinates and 
camera pose for features within the frames; 

extracting virtual key frames from each partial model, the virtual key frames having three- 
dimensional coordinates for the frames and an uncertainty associated with the frames; and 

bundle adjusting the virtual key frames to obtain a complete three-dimensional reconstruction 
of the two-dimensional frames. 

37. (Currently amended) An apparatus for recovering a three-dimensional scene from a 
sequence of two-dimensional frames by segmenting the frames, comprising: 

means for capturing two-dimensional images; 

means for dividing the sequence into s e gm e nts sections, the sections comprising groups of 

frames ; 

means for calculating a partial model for each s e gm e nt section that includes three- 
dimensional coordinates and camera pose for features within the frames; 

means for extracting virtual key frames from each partial model; and 
means for bundle adjusting the virtual key frames to obtain a complete three-dimensional 
reconstruction of the two-dimensional frames. 
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